
REMARKS 

In the office action mailed March 10, 2003, claims 1, 2 and 17 are rejected under 35 
USC § 1 12, second paragraph, as being indefinite for failing to particularly point out and distinctly 
claim the subject matter which Applicant regards as the invention. Claims 1-6, and 1 1-16 are 
rejected under 35 USC § 103(a) as being unpatentable over Elliott et al. (U.S. patent 6,431,875 Bl) 
in view of Stuppy (US Patent 6,146,148). Claims 8 and 18 are rejected under 35 USC §103(a) as 
being unpatentable over Elliott et al. in view of Stuppy and in view of Mishkin (U.S. Patent 
6,377,781 Bl). Finally, claims 9, 10, 19 and 20 are rejected under 35 USC § 103(a) as being 
unpatentable over Elliott et al. in view of Stuppy in view of Siefert (U.S. Patent 6,386,883 B2). 

In response to the rejection of claims 1 and 2 as being directed to a method of 
evaluating group-administered tests but not setting forth a step of evaluating class profiles, 
Applicant respectfully requests reconsideration. However, each of the claims includes at least one 
step which would be considered an evaluation step. Namely, claim 1 includes a step of "comparing 
^each class profile with the normative profile on a question by question basis." Similarly, claim 2 
includes a step of "identifying individual ones of the class profiles that excessively deviate from the 
normative profile on a question-by-question basis." Applicant submits that such steps would 
broadly be considered evaluation steps, and therefore the claims are not ambiguous. 

In response to the rejection of claim 11, Applicants have amended the claim to more 
clearly claim a step of providing a set of sub-group profiles wherein each sub-group profile 
comprises a plurality of individuals whose test item responses are exposed to "a common element 
affecting test administration." Applicant respectfully submits that the claim is no longer ambiguous. 

RESPONSE TO REJECTIONS UNDER 35 USC §103(a) 

The evaluation of the present Application requires differentiating the expected 
variation among test results, which is common in all test settings, and the unexpected variation 
identified by the Applicant's invention. The inventor has looked for variation in an area where it 
should not exist and found it, by the method of the invention, by holding the normal, expected 
variation constant. As a consequence, when an unexpected intrusion of an improper influence on 
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the test administration is present, the invention of this application is uniquely poised to detect it. 
The following information should make this point clear. 

Individual test results are expected to vary. The entire purpose of testing is to 
devise instruments (tests) on which individual performance varies and to then measure individual 
results against the pattern of variation established by other individuals who take the same test 
(i.e., Elliott, Col. 2 Lines 35-42 & 60-67) or against performance standards (i.e., Siefert, Col. 8, 
Lines 7-24; Brown et al., Col. 4, Lines 3-6). When test developers devise their instruments, they 
are equally determined to eliminate variation due to non-test characteristics. Thus, for example, 
the reading requirements are minimized on math tests and all test takers are given the same set of 
test instructions. Over the length of the twentieth century, test developers have improved their art 
to create test content more accurately focused to its intended purpose, to construct the test 
materials or software to represent the content without changing its difficulty or adding additional 
constraints, and to specify the test instructions and administration to provide each test taker with 
the same, neutral opportunity to express his or her capabilities. All this with the single purpose to 
ensure that variation in the test results reflects individual differences on the test construct. 

Just as individual, and even group, test scores are expected to vary with the 
capabilities of the test takers, the pattern of internal test performance that sums up to their test 
results is expected to be consistent. Relatively difficult questions are expected to be relatively 
difficult for all test takers and in all test taker groups (i.e. classrooms). The relative difficulty of 
each test question is reflected in the percentage of test takers who answer the question correctly. 
Easy question: high percent correct; difficult question: low percent correct. These question by 
question percentages may be assembled, over all the questions of the test, to make up a group or 
classroom profile. Anyone profile should be unremarkable, other than the percentages, in 
general, may be higher or lower depending on the skill level of the group. Nevertheless, the 
percentages should rise at the easier questions and fall at the more difficult questions in a similar 
pattern for all groups. The pattern of this profile should be consistent even when the test scores, 
by individual or by group, vary. This consistency underlies test score reliability. The power of 
this consistency, in the presence of normal, construct related variation, to set a norm and 
illuminate instances of improper influence is clearly unanticipated by the prior art. 
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In each of the references to the prior art cited by the prior review of this 
application, the unit of analysis is student or test taker behavior and the references to norms or 
standards are to normative student or test taker behaviors (i.e., Mishkin, Col. 7, Lines 44-47; 
Siefert, Col. 8, Lines 61-64 & Col. 11, Lines 27-33; Stuppy, Col. 10, Lines 41-49). In nearly all 
such references, these norms are expected to reflect student or test taker variation on the 
construct of the test. In the case of Elliott, there are suggestions of norms of test takers who cheat 
and of test taker response times (Elliott, Col 18 Lines 23-48). In all of the cited references, there 
is no indication of potential variation in test results due to the test administrator or due to a 
common element affecting test administration. Quite the contrary, Elliott credits test 
administrators as guarding against variations due to cheating (Elliott, Col. 17, Lines 16-20). 

In the invention of the Applicant, the unit of analysis is the question-by-question 
pattern of group behavior under the direction of a test administrator or subject to a common 
element affecting test administration. In such groups, individual behavior is subsumed to the 
group. Groups may vary in skill level (the construct of the test) such that higher skilled groups 
will achieve a higher success rate (higher percents correct) on the test and lower skilled groups a 
lower success rate (lower percents correct). But, the pattern of relatively higher and lower 
percents correct within the profile, on a question-by-question basis, will remain essentially the 
same for both higher and lower skill groups. Significant variation in the pattern among group 
profiles, on a question-by-question basis, will most likely be due to variations in behavior 
initiated by the test administrator, not by the students or test takers. 

Thus, while the methods of organization of data or of statistical analysis may be 
applied to both the norms of student behavior and the norms of test administrator behavior, the 
behaviors are categorically different and unrelated. The great gap between them is evidenced by 
the large number of instances of alleged teacher cheating, the earnest efforts by experts in the 
field to confirm or disconfirm the allegations, and yet the failure of any prior development of the 
method of the Applicant's invention. The inventor has applied the method of the invention in a 
large number of test administration reviews. These reviews have confirmed the normal 
consistency of group test item response patterns (group or classroom profiles) and the sharp 
divergence that occurs when an improper influence has been applied. 
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In response to the rejection of Claims 1-6, and 1 1-16 under 35 USC §103(a) as being 
unpatentable over Elliott et al. (U.S. patent 6,431,875 Bl) in view of Stuppy, Applicants 
respectfully submit that the claims are allowable over a combination of Elliott and Stuppy. As set 
forth in Applicant's response mailed August 11, 2003, Elliott fails to disclose a set of class profiles 
by test administrator, or generating a normative profile indicative of normative class performance 
on the selected test, where the members of each class included in the normative profile are 
subject to the direction of a test administrator on the selected test. Unlike Applicant's invention 
which addresses generating class profiles by test administrator, as set forth in claims 1 and 2 for \/ 
example, Elliott addresses problems related to tests administered over the internet. For example, 
Elliot suggests using certain decoy questions and response times to detect cheating. Stuppy is cited 
for teaching a system "developing student profiles on a battery of tests that are subject to the 
direction of a computer administrator, that the tests are scored and norm comparisons are made, and 
that student results are tracked and managed by the computer administrator." However, Stuppy is 
directed to a system for tracking individual students performance against a norm, not (i) comparing 
each of the class profiles with a normative profile, as required in claim 1, (ii) identifying individual 
ones of the class profiles that deviate excessively from the normative profiles on a question-by- 
question basis, as required in claim2, or (iii) evaluating administration of the selected test based on 
the comparison of each of the sub-group profiles with the normative profiles on a question by 
question basis, as required in claim 1 1 . 

In particular, Stuppy discloses a system where a student takes the assessment test 
and answers electronically using the pen or other input device, and the answers are scored 
electronically so that the student's initial assessment is free of scoring and administration errors. 
The student may also receive a battery of tests which are generally designed to identify the 
ability to perform different tasks or the mastery of certain learning objectives or skills. The 
assessment tests are scored and analyzed by computer to generate a student profile which is then 
utilized by the system of Stuppy to generate a learning program suited to that student and deliver 
electronic student workbooks. The student profile contains, among other data, skill gaps which 
need to be filled by further instruction. Ongoing assessments during the use of the system are 
used to expand and update the profile. (Col. 4, lines 52-57). However, neither reference 
discloses any evaluation based upon class profiles or sub-group profiles as claimed by Applicant. 
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Accordingly, Applicant submits that any combination would not lead to Applicant's invention as 
claimed. 

In response to the rejection of Claims 8 and 18 under 35 USC § 103(a) as being 
unpatentable over Elliott et al. in view of Stuppy and in view of Mishkin, Applicant respectfully 
submits that these claims are clearly allowable over the cited art for the same reason that 
independent claims 1, 2 and 1 1 are believed allowable. Mishkin is cited as teaching that individual 
profiles may be accumulated into larger units as defined by sessions for entire classes and groups of 
classes, including any sub-grouping desired based upon the number of sessions taking the same 
quiz. Mishkin suggests accumulating "quiz data" (Col. 4, 50-59) and fails to disclose providing a 
set of class profiles by test administrator or providing sub-groups comprising a plurality of 
individuals whose test item responses are exposed to a common external influence as claimed by 
Applicant. Accordingly, any combination of the references would not lead to Applicant's 
invention for the same reasons set forth above. 

Finally, in response to the rejection of Claims 9, 10, 19 and 20 under 35 USC 
§ 103(a) as being unpatentable over Elliott et al. in view of Stuppy and in view of Siefert, Applicant 
also respectfully submits that these claims are clearly allowable over the cited art for the same 
reason that independent claims 1, 2 and 1 1 are believed allowable. Siefert, which is directed to a 
computer assisted education program, also fails to disclose providing a set of class profiles by test 
administrator or providing sub-groups comprising a plurality of individuals whose test item 
responses are exposed to a common element affecting test administration, as claimed by 
Applicant. Accordingly, any combination of the references would not lead to Applicant's 
invention for the same reasons set forth above. 

In view of the above amendments and remarks, Applicant respectfully requests 
reconsideration of the claims. 

Respectfully submitted, 
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